Papers with speech LMs
From Tower to Spire: Adding the Speech Modality to a Translation-Specialist LLM (2025.findings-emnlp)
Copied to clipboard
Kshitij Ambilduke, Ben Peters, Sonal Sannigrahi, Anil Keshwani, Tsz Kin Lam, Bruno Martins, Andre Martins, Marcely Zanon Boito
| Challenge: | Spire is a speech-augmented language model capable of translating speech input into 10 languages and transcribing text input in both directions. |
| Approach: | They introduce a speech-augmented language model capable of translating speech input into 10 languages . they integrate the model into existing multilingual LMs via speech discretization . |
| Outcome: | Spire integrates speech-augmented language model into existing multilingual model using speech discretization and pre-training using only 42.5 K hours of speech. |